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ABSTRACT 


Producing quality digital educational content is the goal of any teaching/learning system, including the teaching/leaning 
technologies contexts. This paper introduces how the Quantitative Evaluation Framework may be used to accurately 
evaluate the digital content produced throughout the development process of an inclusive and integrative MOOC in 
Educational Technologies, based on a Postgraduate Degree from Instituto Superior de Engenharia do Porto (School of 
Engineering at the Polytechnic Institute of Porto). The model is sustained in the SCORM standards, adopting the ISO 
9126 standard as a reference, and it proposes a quantitative representation in an orthogonal three-dimensional 
environment: pedagogical, ergonomics, and management. Each of these domains comprises a set of factors for which it is 
quite relevant to determine the degree of the system’s performance. The factor is a component that represents the 
performance degree of the digital content in the system, following a predetermined quality criterion. The measurement of 
the quality of the educational digital contents is assured by quantitative methods, similar to the evaluation techniques 
commonly used in the education system, which makes its application simpler. 
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1. INTRODUCTION 


Considering the History of Education in Portugal, grammar and high school teachers develop their 
professional activity in accordance with the fundamental principles consigned in the Portuguese Constitution 
and in both the general and specific principles of the Ministry of Education (ECD, 2010), which imply the 
constant need for teachers to be professionally up-to-date not only regarding the contents in their knowledge 
areas but also concerning the pedagogical approaches. Therefore, it is extremely relevant to invest in this 
continuous update. 

Having that need in mind, a postgraduation sustained in an innovative concept was conceived and made 
available targeting at the professionals in Education already in practice or concluding undergraduation 
(Marques & Escudeiro, 2016). That postgraduation is divided into two different courses, both complying 
with the needs of the professionals in Education: a postgraduation in Supporting Technologies for Education 
(TAE), and a postgraduation in Informatics in Education (IE). Their particularity and uniqueness concern 
their modular working structure enabling a flexible enrolment of the participants, as presented in Marques 
& Escudeiro (2016), and available in ESTAE, 2012. 

After realizing the limitations of the blended learning model to reach the target audience widely, an 
inclusive and innovating pedagogical model started being devised, having as premise the at distant 
educational model known as Massive Open Online Course (MOOC). This pedagogical model will enable the 
development of a course model based on the learning process (versus accreditation), making use of Web 
tools which allow any person holding minimal computer user’s expertise to enlarge and/or improve their 
knowledge in a certain topic, or to learn new contents (Escudeiro, 2016), (Marques et al., 2017). 
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The development of this Postgraduation in a MOOC, relying on internet connectivity, widens the possible 
participants to unlimited numbers, opening the access to knowledge in any part of the Planet with no time 
limits — people anywhere have access to learning by the principles of the called distance education and open 
education. 

In these specific MOOCs it is intended to create an inclusive model which will allow the participation of 
deaf and of blind people. Hence, the authors are innovating in the Educational field. 

In order to assure the postgraduation degrees’ homogeneous features, a model comprising a set of 
recommendations grounded in a pedagogical structured supported by an online adapted technology is being 
devised, i.e. a common model to every MOOC. Within this format, any content production must consider the 
following decisive factors: structure, length, pedagogical design, content production and validation 
(Escudeiro, 2016). 

Following the creation of a MOOC pedagogical model in Educational Technologies, this paper presents 
the quality and evaluation model applied to guarantee the quality control of all the digital content produced. 
This quality model is called Quantitative Evaluation Framework (QEF), and it applies a survey based 
analysis as well as an example of a MOOC model (Escudeiro & Bidarra, 2006). 


2. MOOC PEDAGOGICAL MODEL 


The development of the MOOC pedagogical model introduced in this paper comprehends two Postgraduation 
degrees: The Informatics in Education (IE) and the Supporting Technologies for Education (TAE). In the 
subsections that follow both the implemented pedagogical model and the designed solution are described. 


2.1 Implemented Pedagogical Model 


The proposed pedagogical model implies at its founding the involvement of multidisciplinary teams, 
constituted by professionals with distinct backgrounds, so that an accurate educational environment may be 
built comprising an accessible, inclusive and innovative pedagogical setting. 

Planning is the fundamental and grounding task, common to all the individuals involved in the 
development process. Throughout the conception of the pedagogical model, there was especial focus on the 
team to guarantee the active participation of all individuals in accordance with the objectives previously set 
and the defined plan. The cohesion and compromise of the team enable a consistent development which will 
lead to a product totally adapted to a specific target as presented in Figure 1. 

As for the technical aspects, the computer team in the conceptual schematics (Figure 1) represents the 
team of programmers who are responsible for developing the Sign language application that will enable the 
deaf and hearing impaired to have access to the digital content as well. This application, named VirtualSign 
(GILT, 2016), particularly used in educational contexts allows the deaf and the hearing impaired to have 
access to both digital educational contents and digital learning objects. 

The MOOC design is the design team’s responsibility. This team is, therefore, responsible for the 
storyboards which guide the alignment of the lessons’ recordings so that flaws in the planning and tasks 
performance may be avoided. The design team is also in charge of video recording, image and audio editing, 
animations and graphics integration, closely following the contents selected and aligned by the specific 
responsible teacher. 
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Figure 1. Active intervenients in the innovative MOOC pedagogical model 


The provider is responsible for disseminating the final result of this MOOC pedagogical model. 
Dissemination will take place within the provider’s network of contacts, thus promoting the postgraduation 
degrees. 

As for the participants, the target audience of the final product, the model also considers them, once they 
are the ones who will actually use the educational product to be made available in the provider’s platform. 

The teachers are responsible for planning and developing the contents, in which written texts (direct 
speech), images, videos (tutorials, for example), among others are included. Afterwards, the structure of the 
lessons is validated by the coordination team responsible for each course unit, and/or by the responsible 
teacher (in this case the design team is in charge of correcting and adjusting the audio only to guarantee the 
sound quality). 

The research team (Investigators, in Figure 1) is responsible for the research on the blind/visually 
impaired and the deaf/hearing impaired people’s specific needs so that the digital content is provided 
appropriately in the platform enable a fluid and intuitive interaction. 

Finally, the project management team, in addition to supervising the planning, provides technological and 
scientific support to the project, thus contributing for a successful development. 
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2.2 Implemented Solution 


The proposed solution for the pedagogical model implementation is described in this section, where it is also 
possible to understand how the inclusion of the digital contents to be accessed also by the deaf/hearing 
impaired and the blind/visually impaired is considered. 

In Figure 2 each element represents a fundamental role in the implementation structure: the digital content 
repository (DB), where all the digital materials concerning each course unit’s lessons are stored, the 
responsible(s) for content validation, the content experts (teachers), who may use diverse tools as support to 
provide their contents (Word, PowerPoint, video, among others), the Kinect and the Sensor gloves the 
equipment which enables feeding the system with the Portuguese Sign Language specifications to be read by 
the computer, the avatar incorporated in the material to be made available to the students, which allows the 
output of written text translated into Sign Language. 
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Figure 2. Design Solution 
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The MOOC developed based on the proposed pedagogical model must be clear, objective and 
user-friendly for all, inclusive of the deaf/hearing impaired and the blind/visually impaired as already stated. 
It is expected that the participants are provided with an environment that allows them to clearly understand 
how to interact with the contents so that they may succeed in accomplishing the tasks. 

In order to guarantee the effective success of this MOOC pedagogical model, the Quantitative Evaluation 
Framework (QEF) was selected. This evaluation framework will enable controlling the quality of the final 
product by controlling its development process. 


3. QEF IN DIGITAL CONTENT EVALUATION 


Clearly indicate advantages, limitations and possible applications. Following the creation of the MOOC 
pedagogical model in Educational Technologies, this section presents the quality and evaluation model used 
to guarantee the quality control of the digital content produced. This quality model - the Quantitative 
Evaluation Framework - applies a survey-based analysis and a MOOC model example (Escudeiro & Bidarra, 
2006). What is more, the quality and evaluation model for quality control of the digital content proposed in 
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this paper is sustained in software engineering objectives, principles and actions set for evaluating digital 
content. 

The educational branch of software engineering integrates processes, methods and tools in developing 
evaluation models for educational contents aiming to improve their quality. This quality and evaluation 
model may be generally applied in the development of contents for digital systems in order to validate and 
evaluate them throughout their development cycle in any of its phases, therefore enabling the immediate 
detection and correction of possible flaws. 

The model complies with the SCORM standards and references to the (ISO 9126) standard, proposing a 
quantitative representation in an orthogonal three-dimensional environment involving the pedagogical, the 
ergonomics, and the management domains. Each of these domains comprises a set of factors for which it is 
quite relevant to determine the performance degree within the system, being the factor a component that 
represents the performance degree of the digital content in the system, following a predetermined quality 
criterion (Escudeiro & Bidarra, 2008). 


3.1 The dimensions of Digital Contents 


The pedagogical dimension is mainly supported in the learning. Learning is determined by several factors 
which imply the interrelation between the individual, usually identified as student/participant, and the object, 
identified as the pedagogically featured technological instrument. In this context, the evaluation is an 
instrument in the pedagogical practice that allows to verify which are the most valid technological procedures 
in pursuing the educational objectives (Bloom). Within the MOOC pedagogical model in Educational 
Technologies, in the pedagogical domain, two factors were considered: learning and evaluation. The quality 
requirements/criteria associated to each factor in this domain are represented in Table 1. Each quality 
requirement/criterion proposed identifies the metrics, thoroughly selected, which will allow a quantitative 
assessment of the dimension it refers to. 

The ergonomics domain handles the human being’s scientific knowledge and its conception when 
applying it to building equipment and tools that will guarantee the global performance in a certain 
educational system. This domain assures that those tools and equipment are used to promote comfort and 
safety, that is, that they have the required conditions to their appropriate use in the learning environments. 
For the ergonomics dimension, the pedagogical model of the MOOC in Educational Technologies has 
considered the following factors: usability, video/audio, and text. In this intermediate stage the Ergonomics 
dimension of our quality environment considers those factors to be essential as they imply the easy use of the 
interface without disregarding the interaction with its functions within the system. In other words, they refer 
to the degree with which the participant is allowed to perform the tasks efficiently (Escudeiro & Bidarra, 
2008). The requirements/criteria relating to the usability, video/audio, and text factors in the Ergonomics 
domain are detailed in Table 1. 

The Management domain dimension references to the digital educational resources that clearly allow to 
explore the unique characteristics of technology, promoting learning processes that cannot be developed 
using other conventional means, therefore promoting innovation in digital educational resources. This 
dimension reflects the management characteristics of the educational digital contents as regards the 
operational aspects. The intermediate stage of the Management dimension of the quality environment we are 
proposing congregates two factors: content management and adaptability. The quality requirements/criteria 
of the Management domain are detailed in Table | as well. 
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Table 1. The Dimensions applying the Quantitative Evaluation Framework 


Requirements 


- Contents must be hierarchically and sequentially planned. 

- Contents must be divided into several knowledge stages, always starting in the least 
complex stage. 

- Contents, the course unit core, must reflect the best scientific or pedagogical evidence 
available conceming the subjects to be handled, and must be internally coherent, i.e. the 
considered subjects have to be clearly linked and interconnected. 

- In each lesson/video class the interaction with the participanU/attendce must be considered 
by including content-related questions directly addressed to the participant/attendee. 

- A course unit must provide constructive feedback. 

- A course unit must be prepared for participants with different profiles/disabilities. 


~ A course unit must provide problems to be solved in a short penod of time. 

- The activities proposed in the course unit must consider the participants’ collaborative 
work and skills, 

~ The course unit must propose critical reflections about its contents and developed 
assignments. 

- The course unit must allow the participants to choose their path while attending it. 

~ The course unit must promote interactions and foster team work. 

~ The Special Education course units (deaf and blind) comprising specific scientific and 
pedagogical contents must be validated by experts in these fields. 


~ The participant must be able to start and conclude each lesson when he/she wishes it, 

- The course unit must provide help through complementary material. 

+ The lesson’s complementary material must be of easy and intuitive access. 

~ The course units must consider a uniform help patter, 

- The course unit must have various audios available, compliable with the participant's 
needs (including the blind participants). 

~ The course unit must allow the participant to configure the audio, 

~ The deaf must have access to the digital content by means of an automatic bidirectional 
translator which translates the Portuguese written language into the Portuguese Sign 
Language. 

~ The system must have an avatar to foster the interaction with the deaf participants, 

- A help button must be available for the deaf/hearing impaired. 

- A help button must be made available for the blind/visually impaired. 

~ The lesson must use color combination appropriately (accessibility), 

- The lesson must make use of visual resources such as images and icons, in order to help 
transmitting the content better. 

~ The lesson must consider human perception, i.¢, must be prepared for the diverse 
participants” physical abilities/capabilities (deaf/blind), 

- The course must enable the participant to receive feedback in a forum. 


- The course is supported in digital video classes. 

- Video classes must have 8 to 10 minutes-length, corresponding to each lesson. 

~ Each course unit nuust have a brief introduction to the lessons. 

~ In the video edition, the use of images, graphics and animations must be specifically 
prepared for the blind/ visually impaired by a detailed audio description. 

~ The audio is recorded in Portuguese. 

~ The whole text ts presented in linear and concise form. 

- The text included is written in Portuguese. 


~ The lesson title must be clear, objective and appropriate to the content. 
- The content must be written following the Portuguese spelling agreement. 
- Whenever references are used, these have to be included in the bibliography. 


~ Contents are created by a team of certified experts in the field of knowledge. 

~ There is a previous and appropriate content planning to assure the courses homogencous 
features. 

- Contents must be validated by the course unit's responsible teacher. 

~ Contents addressing the blind/visually impaired must be validated by experts in the field, 
- Contents addressing the deaf/hearing impaired must be validated by experts in the field. 
- Contents must be certified by the appropriately certified entities. 

- The course is adapted to be attended by the deaf/hearing impaired by integrating the 3D 
avatar, 

~ The course is adapted to be attended by the blind/visually impaired by audio analysis and 
processing. 
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4. CONCLUSION 


This paper proposed a qualitative model sustained in metrics that enable the quantitative measuring of the 
quality of a specific digital educational content in the context of the pedagogical model supporting the 
MOOC in Educational Technologies. The QEF — Quantitative Evaluation Framework — model, will be the 
basis for supporting, validating, evaluating, controlling and guaranteeing the quality in creating the digital 
educational contents. It will allow predicting the deviations in relation to the initial specifications, even 
before they are included in the educational course applied to the MOOC format. This method is being used in 
the development of the educational contents of the course units comprised in the Postgraduation Degrees of 
Educational Supporting Technologies at Instituto Superior de Engenharia do Porto. 

It is important to highlight that the measurement of the quality of the digital educational content that this 
model allows requires is assured by quantitative methods, similar to the evaluation techniques commonly 
used in the Portuguese educational system, being therefore known by all the intervenient in the 
teaching/learning process, which makes its application quite simpler. As for the teaching technologies, 
producing quality digital educational content is the core purpose of any teaching/learning system. Hence, this 
model has been created making it possible to closely follow the development process of the digital content 
and, thus, control its production throughout the whole cycle. The QEF model may be applied in the 
evaluation of the development of any educational system, favouring from the direct comparison with 
different educational environments (Escudeiro & Bidarra, 2008). 
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